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Abstract 


Many  scientific  problems  require  sampled  functions  and  analytically  defined  objects  to  appear 
together  in  a  single  visualization.  Volume  rendering  is  a  relatively  new  technique  for  visualizing 
sampled  scalar  functions  of  three  spatial  dimensions  by  computing  2D  projections  of  a  colored 
semi-transparent  gel.  This  paper  addresses  the  problem  of  extending  volume  rendering  to  handle 
polygonal  objects.  Two  solutions  are  presented.  The  first  employs  a  hybrid  ray  tracer  capable  of 
handling  both  geometric  and  volumetric  data.  The  second  consists  of  3D  scan-converting  the 
geometric  primitives  into  the  volumetrk:  dataset  and  rendering  the  resulting  ensemble.  Both  solu¬ 
tions  are  d^gned  to  be  efficient  and  to  avoid  aliasing  artifacts.  Their  relative  costs,  image  quality, 
and  versatility  are  compared  using  examples  from  two  application  areas:  medical  imaging  and 
molecular  graphics.  Techniques  for  adding  shadows  and  texture  are  also  presented. 


1.  Introduction 

As  visualization  techniques  move  from  computer  graphics  laboratories  into  scientific  discip¬ 
lines,  they  are  judged  not  by  standards  of  photo-realism,  but  by  their  ability  to  produce  scientific 
insight.  For  some  problems,  presentation  of  a  single  type  of  data  suffices.  For  others,  additional 
insight  is  gained  by  combining  more  than  one  type  of  data  in  a  single  visualization.  This  paper 
addresses  the  problem  of  rendering  mixtures  of  analytically  defined  objects  attd  sampled  sc^ar 
functions  of  three  spatial  dimensions. 

Most  previous  efforts  in  this  area  have  employed  polygonal  meshes  or  binary  voxel  represen¬ 
tations.  I*izer  et  al.  [1986]  crack  edges  on  each  slice  of  a  computed  tomography  (CT)  dataset  to 
yield  a  set  of  contours,  tile  between  contours  on  adjacent  slices,  supplement  the  resulting  polygonal 
mesh  with  analytically  defined  objects,  and  render  the  ensemble  using  conventional  hidden-surface 
algorithms.  Lorsensen  and  Cline  [1987]  apply  surface  detectors  at  each  sample  location  to  produce 
a  large  collection  of  voxel-sized  polygons  which  can  be  supplemented  with  analytically  defined 
objects  and  rendered  using  conventional  algoritiims.  Kaufman  (with  others)  threshold  volumetric 
data  to  produce  a  binary  voxel  representation,  use  3D  scan-conversion  to  add  polygons  [1987b]. 
polyhedra  [1986],  and  cubic  parametric  curves,  surfaces,  and  volumes  [1987a]  to  the  array,  and 
render  the  resulting  ensemble  using  custom-designed  hardware  [1988a]. 

All  of  these  approaches  require  a  binary  classification  of  the  volumetric  data.  As  a  result, 
they  suffer  from  false  positives  (spurious  objects)  and  false  negatives  (missing  objects),  particularly 
in  the  presence  of  small  or  poorly  defined  features.  Kaufman’s  use  cf  a  binary  voxei  representation 
fer  analytically  defined  objects  gives  nse  to  additional  artifacts,  although  a  solution  to  this  problem 
has  apparently  been  worked  out  [I988bj. 
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The  approach  taken  in  this  paper  is  based  on  a  visualization  technique  called  volume  render¬ 
ing^  in  which  a  color  and  an  opacity  is  assigned  to  each  voxel,  and  a  2D  projection  of  the  resulting 
colored  semi-transparent  gel  is  computed  [Levoy.  1988a,  Drebin  et  al.,  1988,  Sabella,  1988,  Upson 
and  Keeler,  1988].  The  principal  advantages  of  volume  rendering  over  other  visualization  tech¬ 
niques  are  its  superior  image  quality  and  its  ability  to  generate  images  without  explicitly  defining 
surfaces. 

Although  many  researchers  have  suggested  methods  for  extending  volume  rendering  to  han¬ 
dle  analytically  defined  objects,  no  implementations  have  yet  appeared  in  the  literature.  Sun 
Microsystems  has  reportedly  extended  a  conventional  ray  tracer  to  handle  volumetric  data  [Mosher, 
1988],  but  their  algorithm  has  not  been  published.  In  an  paper  currently  in  review,  Goodsell  et  al. 
[19881  describe  a  two-pass  approach  tlun  combines  a  Z-buffer  algorithm  for  rendering  polygonally 
defined  atomic  structure  and  a  ray  tracer  for  rendering  volumetric  molecular  data.  Although  their 
method  produces  satisfactory  visualizations  in  many  cases,  it  cannot  efficiently  handle  semi¬ 
transparent  polygons.  Furthermore,  the  division  of  later  into  two  passes  necessitates  rendering  all 
polygons  even  though  they  might  be  obscured  by  volumetric  data. 

This  paper  presents  two  methods  for  extending  the  volume  rendering  algorithm  described  by 
Levoy  [1988a]  to  handle  polygonally  defined  objects.  The  first  method  employs  a  hybrid  ray 
tracer.  Rays  are  simultaneously  cast  through  the  volumetric  sample  array  and  the  polygonal  data¬ 
base,  samples  of  each  are  drawn  at  equally  spaced  intervals  along  the  rays,  and  the  resulting  colors 
and  opacities  are  composited  together  in  depth-sorted  order.  To  avoid  errors  in  visibility, 
volumetric  samples  lying  immediately  in  firont  of  and  behind  polygons  are  given  special  treatment 
To  avoid  aliasing  of  polygonal  edges,  adaptive  supersampling  is  used.  The  second  method  employs 
3D  scan-conversion  with  analytic  anti-aliasing.  Polygons  are  shaded,  filtered,  sampled,  and  com¬ 
bined  with  the  volumetric  data.  The  resulting  composite  dataset  can  then  be  rendered  using  pub¬ 
lished  methods.  If  the  polygonal  data  is  sufficiently  bandlimited  prior  to  sampling,  this  method  also 
produces  images  firee  from  aliasing  artifacts. 

To  compare  the  relative  versatility  of  these  two  methods,  techniques  for  adding  shadows  and 
textures  will  jdso  be  considered.  Nelson  Max  has  written  a  brief  but  excellent  survey  of  algorithms 
for  casting  shadows  [1986].  The  present  study  employs  a  two-pass  approach  [Williams,  1978],  but 
stores  shadow  information  in  a  3D  light  strength  buffer  instead  of  a  2D  shadow  depth  buffer.  The 
amount  of  memory  required  for  a  3D  buffo'  is  obviously  much  greater,  but  the  representation  has 
several  advantages.  By  computing  a  fractional  light  strength  at  every  point  in  space,  penumbras 
and  shadows  cast  by  semi-transparent  objects  are  correctly  rendered.  Moreover,  the  shadow  alias¬ 
ing  problem  encountered  by  Williams  does  not  occur.  Finally,  the  present  algorithm  correctly  han¬ 
dles  shadows  cast  by  volumetrically  defined  objects  on  themselves,  as  well  as  shadows  cast  by 
polygons  on  volumetric  objects  and  vice  versa. 

There  are  several  kinds  of  texture  mapping  that  might  be  useful  when  rendering  mixtures  of 
geometric  and  volumetric  data.  Wrapping  textures  around  volumetrically  defined  objects  requires 
knowing  where  their  defining  surfaces  lie  -  a  hard  problem.  Projecting  textures  through  space  and 
onto  these  surfaces  is  much  easier  and  can  be  handled  by  a  straightforward  extension  of  the  shadow 
casting  algorithm.  Mapping  textures  onto  polygons  embedded  in  volumetric  datasets  is  also  rela¬ 
tively  simple.  The  two  latter  techniques  will  te  demonstrated  in  this  paper.  A  good  survey  of  tex¬ 
ture  mapping  algorithms  has  been  written  by  Heckbert  [1986].  The  present  study  employs  an 
approach  similar  to  that  described  by  Feibush.  Levoy,  and  Cook  [1980],  but  since  geometric, 
volumetric,  and  texture  data  each  exhibit  different  spectra,  care  must  be  taken  when  mixing  them. 


t  In  the  Reid  of  imafc  proceising.  (his  tmn  hss  been  used  to  denote  techniques  based  on  oveiptinting  of  opaque  vox¬ 
els.  Ji  computer  giiptucs,  it  has  come  (o  denote  techniques  based  on  blending  of  semi-transparent  voxels.  To  distinguish 
between  these  approaches,  Reynolds  (I988|  has  suggested  that  volume  rendering  be  used  to  encompass  both  of  them  and  the 
more  specific  votumeiric  compositing  be  used  to  denote  the  partial  transparency  variant  To  avoid  confusion,  I  will  continue 
to  use  current  computer  graphics  terminology  until  a  new  one  gains  acceptance  in  the  research  community. 
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2.  Hybrid  ray  tracer 

The  first  rendering  method  we  will  consider  is  the  hybrid  ray  tracer  shown  in  figure  1.  Let 
us  assume  sn  N  x  N  x  N  voxel  array  of  scalar  values  and  a  set  of  embedded  polygons.  We  begin 
by  shading  and  classifying  the  volumetric  data  to  yield  a  color  cy  and  opacity  Oy  for  each  voxel 
[Levoy,  1988a].  Parallel  viewing  rays  are  thra  traced  into  the  data  from  an  observer  position  as 
shown  in  figure  2.  For  each  ray.  a  vector  of  colors  cy  and  opacities  tv  is  computed  by  resampling 
the  volumetric  data  at  equally  spaced  positions  along  the  ray  and  tri-linearly  interpolating  from  the 
colors  and  opacities  in  the  eight  voxels  surrounding  each  sample  location.  Independently,  all  inter¬ 
sections  between  the  ray  and  polygons  in  the  environment  are  computed  and  shaded,  yielding  a 
color  Cf>  and  opacity  for  each  point  of  intersection.  Finally,  the  resampled  volumetric  colors  and 
opacities  are  composited  with  each  other  and  with  the  polygonal  colors  and  opacities  in  depth- 
sorted  order  to  yield  a  color  for  the  ray. 

Volumetric  compositing  coirecdy  renders  the  appearance  of  a  gel  composed  of  many  small 
slabs  each  of  identical  size  and  homogeneous  cotor  and  opacity  as  shown  in  figure  3a.  The  thick¬ 
ness  of  each  slab  is  equal  to  the  spacing  between  samples  along  a  viewing  ray.  and  the  width  of  the 
slab  is  equal  to  the  spacing  between  adjacent  rays.  If  both  spacings  are  set  roughly  equal  to  the 
spacing  between  voxels  in  the  volumetric  data,  and  tri-linear  interpolation  is  used  to  compute  the 
color  and  opacity  of  each  slab  as  described  in  the  previous  paragraph,  the  resulting  image  will  gen¬ 
erally  be  firM  of  aliasing  artifacts. 

When  a  polygon  is  embedded  in  the  volumetric  data  as  shown  in  figure  3b,  it  passes  through 
some  of  these  slabs,  obscuring  some  portion  of  the  gel  in  each  slab  and  being  obscured  by  the 
remainder.  An  exact  solution  of  the  hidden-volume  problem  inside  every  slab  would  be  extraordi¬ 
narily  expensive.  Supersampling  is  an  alternative,  but  also  expensive.  A  solution  of  less  accuracy 
and  expense  that  has  proved  satisfactory  in  practice  is  to  treat  a  polygon  locally  as  a  plane  perpen¬ 
dicular  to  the  ray  and  placed  at  the  point  of  intersection  as  shown  in  figure  3c.  An  exact  solution 
of  the  hidden-volume  problem  for  this  restricted  case  is  simple.  The  law  of  exponential  attenuation 
[Johns,  1983]  states  that  the  opacity  (v  of  a  slab  of  homogeneous  material  is  exponentially  related 
to  its  density  dy  and  thickness  ty  by 

ay=l-e-^'^  (1) 

where  0  <  cv  <  1.  and  Oy-  =  I  signifies  complete  attenuation.  Let  the  thicknesses  of  those  portions 
of  the  slab  lying  immediately  in  front  of  and  behind  an  embedded  plane  perpendicular  to  die  view¬ 
ing  ray  be  denoted  tf  and  tj  respectively.  Solving  equation  (1)  for  opacities  ap  and  in  terms  of 
opacity  ay  and  thicknesses  ty  and  gives 


Of  =  1  -  (1  -  ay)''''’ 

(2a) 

Oa  =  1  -  (1  -  ay)'*"’'. 

(2b) 

Working  from  front  to  back,  we  first  composite  cy  and  ay  into  the  ray,  followed  by  cy  and  ay,  and 
finally  by  cy  and  ay.  If  the  ray  intersects  more  than  one  polygon  within  a  slab,  the  contribution 
made  by  each  polygon  and  each  sliver  of  volumetric  gel  must  be  computed  and  composited 
separately. 

At  polygonal  edgee,  polygon-polygon  'r.tcrscrtions,  and  pv^lygun  shading  highlights,  such  an 
approximation  does  not  suffice  to  prevent  aliasing.  Therefore,  the  number  of  rays  cast  pjer  pixel  is 
increased  in  these  regions  using  a^ptive  supersampling  [Whitted,  1980].  One  difficulty  with  this 
approach  is  distinguishing  whether  the  color  difference  observed  in  a  sample  region  is  due  to 
volumetric  data  or  geometric  data,  since  supersampling  is  only  appropriate  in  the  latter  instance. 
The  problem  is  solved  by  computing  two  additional  colors  during  ray  tracing  as  shown  in  figure  1. 
The  first,  denoted  Cc,  contains  contributions  only  from  polygons  (obtained  by  setting  ay  -  0  for  all 
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volumetric  samples).  The  second,  denoted  Q,  contains  contributions  only  from  polygons,  but 
attenuated  by  passage  through  the  volumetric  data  (obtained  by  setting  =  0  for  ^1  volumetric 
samples).  If  the  range  of  Cx  within  a  sample  region  exceeds  some  e  but  the  range  of  Cq  does  not, 
the  observed  color  difference  is  due  to  spatially  varying  volumetric  data  rather  than  a  geometric 
event  (such  as  a  polygonal  edge).  Supersampling  is  not  required  in  this  case.  If  the  range  of  Cq 
exceeds  e  but  the  range  of  Q  does  not,  the  region  contains  a  geometric  event,  but  that  event  is  hid¬ 
den  from  view  by  overlying  opaque  volumetric  data.  No  supersampling  is  needed  in  this  case 
either.  Only  if  the  ranges  of  Ixith  colors  exceed  c,  signifying  that  the  region  contains  a  visible 
geometric  event  should  more  rays  be  cast 


3.  3D  scan-conversion 

The  second  rendering  method  we  will  consider  is  3D  scan-conversion  of  the  geometric  data 
as  shown  in  figure  4.  We  begin  by  shading  and  classifying  the  volumetric  data  to  yield  a  color  cy 
and  opacity  ay  for  each  voxel.  Independently,  each  polygon  is  shaded,  filtered,  and  sampled  at  the 
resolution  of  the  volumetric  data  to  yield  a  color  cq  and  opacity  Oc  for  each  voxel.  The  polygons 
used  in  the  present  studies  are  of  homogeneous  color  and  are  filtered  using  a  3D  version  of  the 
edge  filtering  algorithm  described  by  Feibush  et  al.  [1980].  The  convolution  kernel  employed  is  a 
constant  weight  bail  2  voxels  in  dimeter.  Methods  that  are  faster  and  allow  continuous  shading 
are  currently  under  investigation. 

These  two  sets  of  colors  and  opacities  are  combined  using  volume  matting  operators  [Drebin 
et  aL,  1988]  to  yield  a  composite  color  cc  and  opacity  Oc  for  each  voxel.  Since  matting  is  not 
commutative,  the  order  in  which  polygons  are  scan-converted  affects  the  appearance  of  any 
polygon-polygon  intersections.  Polygons  may  also  be  matted  over  or  under  volumetric  data,  which 
also  affects  their  appearance.  The  composite  dataset  is  then  ray  traced,  resampled,  and  composited 
without  giving  further  consideration  to  the  geometric  data  it  contains.  Since  the  polygons  have 
already  been  filtered,  an  image  sampling  rate  of  one  ray  per  pixel  usually  suffices  to  prevent  alias¬ 
ing  artifacts,  even  along  polygonal  edges. 


4.  Shadow  calculations 


Although  both  of  the  rendering  methods  described  above  could  be  extended  to  yield  shadow  infor¬ 
mation,  the  pre-filtered  representation  of  polygons  employed  by  the  scan-conversion  method  makes 
it  the  more  convenient  starting  point  Figure  S  summarizes  the  modified  algorithm.  The  first  steps 
are  shading  and  classification  of  the  volumetric  data,  shading  and  scan-conversion  of  the  polygons, 
and  matting  to  yield  for  each  voxel  a  set  of  composite  colors  Ci, .  .  .  .c^,  for  light  sources 
<  €  1,  ...  X  and  a  composite  opacity  Oc-  Parallel  illumination  rays  are  then  cast  into  the  data 
from  each  light  source  as  shown  in  figure  2.  For  each  ray,  we  assume  some  initial  light  strength, 
draw  samples  of  the  volumetric  data  at  equally  spaced  positions  along  the  ray,  compute  an  opacity 
at  each  location  by  tri-linearly  interpolating  firom  the  nearest  eight  voxels,  and  attenuate  the 
strength  of  the  ray  in  proportion  to  the  computed  opacities.  The  set  of  strengths  Pr . Pc^  com¬ 

puted  at  each  sample  position  along  each  ray  are  stored  in  a  3D  light  strength  buffer. 

Using  volumetric  compositing,  attenuation  of  light  strength  along  an  illuminatior.  ray  is  simi¬ 
lar  to  accumulation  of  opacity  along  a  viewing  ray.  The  strength  P,*,  of  a  ray  as  it  leaves  a  sample 
locauon  can  be  computed  from  the  strength  Pj,  as  it  approaches  the  sample  and  the  opacity  a  of 
the  sample  by  the  formula 


P»-  =  P« 


(i-g) 

ij  +  k'jd 


(3) 


where  and  k-^  are  constants  used  in  a  linear  approximation  of  square-law  light  attenuation  and  d 
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is  the  distance  from  the  light  source  to  the  sample  location. 

When  all  color  and  light  strength  volumes  have  been  computed,  a  total  color  cr  for  each 
voxel  is  computed  as  the  weighted  sum 

L 

(4) 

where  the  P,’s  are  obtained  from  the  Pj>*s  by  mapping  from  voxel  indices  into  the  coordinate  sys¬ 
tem  of  light  source  i  and  tri-linearly  interpolating  tom  the  nearest  eight  light  strength  values.  The 
arrays  of  total  color  ct  and  composite  opacity  ot^  are  then  ray  traced,  resampled,  and  composited  as 
usual. 

One  problem  that  arises  when  casting  shadows  through  volumetric  data  is  the  tendency  of 
surfaces  in  the  data,  which  generally  have  a  thickness  of  several  voxels,  to  partially  shadow  them¬ 
selves.  This  holds  true  for  naturally  occurring  surfaces  (such  as  tissue  boundaries)  as  well  as  those 
introduced  using  30  scan-conversion.  The  solution  adopted  in  this  paper  is  to  translate  the  3-D 
light  strength  volume  a  few  voxels  away  tom  the  light  source  before  computing  the  c/s.  While 
this  expedient  necessarily  reduces  the  accuracy  of  shadowing  within  small  objects,  it  avoids  intro¬ 
ducing  distracting  aliasing  artifacts. 


5.  Texture  mapping 

The  pre-hltered  representation  of  polygons  that  made  the  scan-conversion  method  well  suited 
to  shadow  casting  would  severely  blur  any  texture  applied  to  them.  For  this  reason,  texture  map¬ 
ping  is  better  handled  by  the  hybrid  ray  tracer.  As  part  of  the  shading  calculations  performed  at 
each  point  of  intersection  between  a  ray  and  a  polygon,  a  mapping  is  performed  from  object  space 
to  texture  definition  space  as  shown  in  figure  2.  The  texture  array  is  then  filtered  and  resampled  at 
that  location.  In  the  present  studies,  the  slow  but  accurate  filtering  algorithm  described  by  Feibush 
et  al.  [1980]  is  employed. 

To  prevent  the  addition  of  textures  from  triggering  the  casting  of  excessive  number  of  rays, 
the  adaptive  supersampling  procedure  described  in  the  last  paragraph  of  section  2  must  be  modified. 
Specifically,  those  aspects  of  the  polygon  shading  model  that  are  analytic  and  might  give  rise  to 
sudden  changes  in  polygon  color  Cf  or  opacity  tv  should  be  included  in  the  supersampling  deci¬ 
sion,  whereas  contributions  by  sampled  functions,  which  are  assumed  bandlimited,  should  noL  In 
the  current  implementation,  this  rule  is  satisfied  by  including  directional  shading  and  depth  cueing 
but  not  texture  mapping  in  the  computation  of  colors  Cq  and  Cj^. 


6.  Optimizations 


The  principal  drawback  of  volume  rendering  techniques  is  their  cost.  Since  all  voxels  partici¬ 
pate  in  the  generation  of  each  image,  rendering  time  grows  linearly  with  the  size  of  the  dataset. 
Fortunately,  these  costs  can  be  reduced  by  taking  advantage  of  various  forms  of  coherence.  Three 
such  optimizations  are  summarized  here  and  described  in  detail  elsewhere  [Levoy,  1988b,  Levoy, 
1988c]. 

The  first  optimuation  is  based  on  the  observation  that  many  datasets  contain  coherent  regions 
of  uninteresting  voxels.  A  voxel  is  defined  as  uninteresting  if  its  opacity  is  zero.  Such  coherence 
is  readily  encoded  as  a  hierarchical  spatial  enumeration  represented  by  a  pyramid  of  binary 
volumes  [Levoy,  1988b].  This  pyramid  is  used  to  efficiently  compute  intersections  between  view¬ 
ing  rays  and  regions  of  interest  in  the  data.  In  the  hybrid  ray  tracer  described  in  section  2,  the 
pyramid  can  be  used  to  speed  the  tracing  of  rays  through  segments  of  volumetric  data  lying 
between  successive  polygon  intersections.  In  the  scan-conversion  method  described  in  section  3, 
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the  composite  dataset  including  the  sampled  polygons  can  be  used  during  construction  of  the 
pyramid.  This  improves  the  efficiency  of  rendering  both  volumetric  and  geometric  data. 

The  second  optimization  is  based  on  the  observation  that  once  a  ray  has  struck  an  opaque 
object  or  has  progiebed  a  sufficient  distance  through  a  semi-transparent  object,  opacity  accumulates 
to  a  level  where  the  color  of  the  ray  stabilizes  and  ray  tracing  can  be  stopped  [Levoy.  1988bl.  In 
the  hybrid  ray  tracer,  adaptive  termination  of  ray  tracing  reduces  both  the  number  voxels  that  must 
be  resampled  and  the  number  of  ray-polygon  intersections  that  must  be  shaded.  Note  that  this 
optimization  must  be  applied  independently  in  the  computation  of  each  of  Cq  and  Q  so  as  not  to 
adversely  aHect  the  adaptive  supersampling  process. 

If  there  is  coherence  present  in  a  dataset,  there  may  also  be  coherence  present  in  its  projec¬ 
tions.  This  is  particularly  true  for  data  acquired  fiom  sensing  devices,  where  the  acquisition  pro¬ 
cess  often  intn^uces  considerable  blurring.  The  third  optimization  takes  advantage  of  this  ct^er- 
ence  by  casting  a  sparse  grid  of  rays,  less  than  one  per  pixel,  and  adaptively  increasing  the  number 
of  rays  in  regions  of  high  image  complexity  (Levoy,  1988c].  Images  may  be  formed  from  the 
resulting  non-uniform  array  of  sample  colors  by  interpolation  and  resampling  at  the  display  resolu¬ 
tion.  This  optimization  is  well  suit^  to  the  scan-conversion  method  in  which  polygons  are  filtered 
prior  to  ray  tracing.  It  may  also  be  integrated  with  the  adaptive  supersampling  employed  in  the 
hybrid  ray  tracer,  but  one  runs  the  risk  of  missing  small  polygons  if  the  initial  sampling  rate  is  too 
low. 

The  cost  of  ray  tracing  volumetric  data  using  a  hierarchical  enumeration  grows  linearly  with 
the  size  of  the  image,  logarithmically  with  the  length  of  each  ray,  and  linearly  with  the  depth  com¬ 
plexity  of  the  data  [Levoy.  1988b].  Adaptive  termination  of  ray  tracing  reduces  the  dependence  on 
depth  complexity,  and  for  the  special  case  of  an  environment  consisting  only  of  opaque  objects, 
eliminates  this  dependence  entirely.  In  this  case,  the  cost  of  generating  an  image  grows  nearly 
linearly  with  the  size  of  the  ima^e.  Adaptive  sampling  further  reduces  this  cost,  and  if  used  care¬ 
fully,  does  so  with  a  negligible  loss  of  image  quality  [Levoy,  1988c]. 

Combining  these  three  optimizations,  savings  of  more  than  two  orders  of  magnitude  over 
brute-force  rendering  methods  have  been  obtained  for  many  datasets.  Alternatively,  the  adaptive 
sampling  method  allows  a  sequence  of  successively  moe  refined  images  to  be  generated  at  equally 
spaced  intervals  of  time  by  casting  more  rays,  adding  the  resulting  colors  to  the  sample  array,  and 
repeating  the  interpolation  and  resampling  steps.  Crude  images  can  often  be  obtained  in  few 
seconds,  followed  by  gradually  better  images  at  intervals  of  a  few  seconds  each,  culminating  in  a 
high  quality  image  in  less  than  a  minute. 


7.  Implementation  and  results 

Both  of  the  rendering  methods  described  in  this  paper  have  been  implemented  in  the  C 
language  on  a  Sun  4/280  with  32MB  of  main  memory.  In  order  to  compare  their  costs  and  image 
quality,  let  us  consider  a  simple  test  environment  consisting  of  thrtre  mutually  perpendicular 
polygons  embedded  in  a  256  x  256  x  113  voxel  computed  tomography  (CT)  study  of  a  human 
head.  Figures  6a  and  6b  are  380  x  380  pixel  views  of  this  data  render^  with  the  hybrid  ray  tracer 
described  in  section  2  and  the  scan-conversion  method  described  in  section  3  respectively. 

The  worst-case  memory  requirements  for  the  optimized  hybrid  ray  tracer  is  2N^  bytes  to  hold 
the  shaded  volumetric  data,  (8^*’*’^*-lV7  bits  to  hold  the  pyramid  of  binary  volumes,  and  a  rela¬ 
tively  small  amount  of  storage  to  hold  the  polygonal  data  and  the  various  image  sample  arrays. 
For  our  test  environment,  this  works  out  to  about  18  megabytes.  In  the  scan-conversion  method, 
polygon  colors  and  opacities  Cc  and  a<;  can  be  matted  with  voxel  colors  and  opacities  cy  and  ay 
on-the-fly,  precluding  the  need  for  additional  storage  during  scan<onversion.  The  worst-case 
memory  requirements  thus  occur  during  ray  tracing  and  are  similar  to  those  for  the  hybrid  ray 
tracer. 
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The  computational  expense  of  shading  and  classifying  volumetric  data  is  proportional  to  the 
number  of  interesting  voxels  it  contains  (voxels  whose  opacity  is  not  zero).  For  this  example,  the 
two  steps  took  about  1  minute  apiece.  The  cost  of  scan-converting  a  polygon  is  proportion^  to  its 
surface  area.  The  large  polygons  used  here  were  scan-converted  in  about  1  second  each.  The  cost 
of  computing  a  pyramid  of  binary  volumes  is  proportional  to  the  number  of  bits  it  contains.  For 
this  example,  pyramid  construction  took  about  1  minute.  In  the  case  of  the  hybrid  ray  tracer,  shad¬ 
ing,  classification,  and  pyramid  construction  are  all  independent  of  both  the  polygon  geometry  and 
the  observer  position,  and  may  thus  be  thought  of  as  pre-processing  expenses.  In  the  scan- 
conversion  method,  the  pyramid  depends  on  the  composite  of  the  volumetric  and  geometric  data 
and  must  re-computed  if  either  changes.  The  remaining  costs  of  image  generation  are  those  of 
adaptive  ray  tracing,  interpolation,  and  resampling,  of  which  the  first  dominates.  The  cost  of  adap¬ 
tive  ray  tracing  is  proportional  to  the  number  of  rays  traced.  Figures  6a  and  6b  were  generated 
using  slightly  fewer  than  one  ray  per  pixel  on  average,  and  took  80  seconds  and  70  seconds  respec¬ 
tively  to  generate.  For  comparison,  the  volumetric  data  without  the  polygons  can  be  rendered  with 
the  same  level  of  refinement  in  about  SO  seconds. 

Figures  7a  and  7b  show  details  from  figures  6a  and  6b.  In  general,  polygon  edges  produced 
by  the  hybrid  ray  tracer  are  sharper  than  those  produced  by  3D  scan-conversion,  although  a  higher 
image  generation  cost  is  paid  for  the  better  rendition.  The  slight  aliasing  noticeable  in  the  bony  tis¬ 
sue  in  these  and  other  figures  is  due  to  insufficient  bandlimiting  during  CT  scanning  and  is  not  a 
result  of  the  rendering  process. 

A  visualization  of  where  rays  were  cast  during  generation  of  figures  6a  and  6b  is  given  in 
figures  8a  and  8b.  Each  pixel  ir.  these  visualizations  corresponds  to  one  position  in  the  array  of 
sample  colors  described  in  paragraph  3  of  section  6,  and  each  4x4  block  of  pixels  corresponds  to 
a  single  image  pixel  in  figure  6.  White  pixels  in  the  visualizations  correspond  to  cast  rays,  and 
black  pixels  to  samples  filled  in  by  interpolation.  As  expected,  the  number  of  rays  per  unit  area  is 
lowest  in  the  interiors  of  homogeneous  regions,  higher  ^ong  the  silhouettes  of  volumetric  objects, 
and  in  the  case  of  the  hybrid  ray  tracer,  higher  still  along  polygonal  edges. 

Figure  9  shows  the  effect  of  casting  shadows  using  the  algorithm  described  in  section  4.  The 
scene  contains  two  light  sources:  a  low-intensity  light  shining  over  the  observer’s  right  shoulder 
and  a  high-intensity  light  shining  up  from  below  and  U>  the  left.  To  insure  that  the  shapes  of  sha¬ 
dowed  objects  are  not  completely  obscured,  shadows  were  only  computed  for  the  high-intensity 
light.  While  this  is  not  strictly  correct,  the  goal  is  enhanced  insight,  not  photo-realism.  Initial  light 
strengths  for  the  high-intensity  source  were  assigned  from  a  texture  containing  a  filtered  rectangular 
grid.  The  effect  is  to  project  this  texture  through  the  dataset  and  onto  all  illuminated  surfaces, 
including  the  five  scan-converted  polygons.  The  addition  of  shadows  roughly  doubles  the  time 
required  to  compute  the  array  of  total  voxel  colors  cj-,  but  docs  not  affect  the  time  required  to  gen¬ 
erate  each  frame  in  a  rotation  sequence  (assuming  fixed  lighting  and  data,  a  moving  observer,  and 
no  specular  highlights). 

Figure  10  shows  the  effect  of  mapping  a  texture  onto  embedded  polygons  using  the  algorithm 
described  in  section  5.  The  effect  on  image  generation  time  of  adding  textures  depends  on  the 
number  and  size  of  textured  polygons.  For  this  example,  image  generation  time  increased  by 
roughly  50%. 

Figure  1 1  suggests  one  possible  way  in  which  these  techniques  might  be  applied  to  the  prob¬ 
lem  of  radiation  treatment  planning.  The  same  CT  dataset  used  in  the  previous  figures  lias  now 
been  rendered  to  show  both  bone  and  soft  tissue.  A  polygonally  defin^  tumor  (in  purple)  and 
radiation  treatment  beam  (in  blue)  have  been  added  using  the  hybrid  ray  tracer.  A  portion  of  the 
CT  data  has  been  clipped  away  to  show  the  3D  relationship  between  the  various  objects. 

Figure  12  illustrates  how  scan-converted  backdrop  planes  and  cast  shadows  can  be  used  to 
enhance  comprehension  of  an  isovalue  contour  surface  from  an  electron  density  map  of  Staphylo¬ 
coccus  Aureus  ribonuclease.  The  polymer  backbone  crosses  the  image  from  bottom  to  top,  and  two 
Tyrosine  residues  with  their  characteristic  six-atom  benzene  rings  can  be  seen  extending  to  the  left 
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and  right  sides  of  the  backbone. 

These  studies  have  led  to  several  welcome  but  unexpected  results.  Polygons  rendered  using 
3D  scan-conversion  appear  to  have  a  finite  thickness  when  rotated.  Their  opacity  also  varies  with 
their  angle  relative  to  the  view  direction,  as  would  a  real  slab  of  semi-transparent  gel.  Far  from 
being  distracting,  these  effects  enhance  the  viewer’s  understanding  of  their  shape  and  orientation. 
The  embedding  of  polygons  in  volumetric  data  also  seems  to  improve  comprehension  of  the  latter. 
For  example,  when  the  CT  study  of  the  head  is  bisected  by  a  gridded,  coronally-oriented  (in  the 
plane  of  the  face)  polygon  and  the  ensemble  is  rotated,  the  presence  of  a  backdrop  of  known  shape 
and  pattern  improves  appreciation  of  the  fine  structure  of  the  sinuses  and  eye  orbits.  This  suggests 
that,  in  addition  to  their  primary  role  representing  man-made  or  abstract  entities,  geometric  primi¬ 
tives  may  be  useful  as  diagnostic  tools  in  the  study  of  volumetric  datasets. 


8.  Conclusions 

Two  methods  for  rendering  mixtures  of  geometric  and  volumetric  data  have  been  described. 
Although  both  are  reasonably  efficient  and  produce  images  free  from  aliasing  artifacts,  each  have 
advantages  and  disadvantages  in  terms  of  cost,  image  quality,  and  versatility. 

A  number  of  improvements  to  these  methods  can  be  suggested.  The  current  implementation 
makes  frequent  use  of  tri-linear  interpolation  for  resampling  3D  data.  A  better  filter  would  reduce 
the  amount  of  blurring  required  during  3D  scan-conversion,  yielding  sharper  polygonal  edges.  In  a 
similar  vein,  the  shadow  casting  algorithm  includes  three  successive  resampling  steps.  By  reorgan¬ 
izing  the  order  of  operations,  one  resampling  can  be  eliminated,  yielding  crisper  shadows.  In  the 
area  of  computational  expense,  the  addition  of  a  separate  hierarchical  data  structure  to  handle 
polygonal  data  would  spe^  up  the  hybrid  ray  tracer.  It  may  also  be  possible  to  use  a  single  data 
strucoire  to  represent  both  geometric  and  volumetric  data. 

Several  strategies  for  treating  mixtures  of  geometric  and  volumetric  data  remain  unexplored. 
For  example,  geometric  objects  defined  by  extrusion  of  2D  profiles  can  be  rendered  using  the  sha¬ 
dow  casting  algorithm.  If  the  profile  is  loaded  into  the  array  of  initial  light  strengths,  and  all  vox¬ 
els  in  the  volumetric  data  are  assigned  a  slightly  non-zero  opacity,  one  obtains  the  effect  of 
sculpted  shafts  of  light  passing  through  a  dust-filled  room.  This  technique  could  be  used  to  visual¬ 
ize  a  radiation  beam  for  caiKer  treatment  If  the  beam  is  assumed  to  pass  through  the  volumetric 
data  without  attenuation,  it  can  be  rendered  without  a  separate  shadow  pass  and  without  the  accom¬ 
panying  3D  representation. 
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Figure  1  -  Overview  of  hybrid  ray  tracer 
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Figure  2  -  Summary  of  arrays  used  during  volume  rendering 


c  -  Approximate  solution  to  hidden-volume  problem 


Rgure  3  -  Rendering  of  polygon  embedded  in  volumetric  data 
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Figure  4  -  Overview  of  3D  scan-conversion  method 
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Figure  5  -  Addition  of  shadow  c3lculations  to  3D  scan-conversion  method 


Figure  6a  -  Volume  rendering  of  human  head  and  embedded  polygons, 
generated  using  hybrid  ray  tracer 


Figure  -"  i:  -  Volume  rendering  of  human  head  and  embedded  polygons, 
generated  using  3D  scan-conversion  method 


Figures  7a  and  7b  -  Details  from  figures  6a  and  6b,  showing  comparative 
image  quality  of  hybrid  ray  tracer  and  3D  scan-conversion  methods 


Figures  8a  and  8b  -  Visualization  of  where  rays  were  cast 
dunng  generation  of  figures  6a  and  6b 
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Figure  10  -  Volume  rendering  with  textured  polygons, 
generated  using  modified  hybrid  ray  tracer 


Figure  1 1  -  Volume  rendering  of  human  head,  showing  bone, 
tissue,  tumor  (purple)  nnd  rndintinn  trontmont  boom  (hluo) 


F  iguro  12  Volume  rendering  of  isovalue  contour  surface 
a  electron  density  map  of  Staphyloccus  Aureus  ribonucleas^^ 


